Time Series Anomaly Detction

1. Import Packages

First, We load the required packages

In [1]:
import pandas as pd
from pandas import ExcelWriter
from pandas import ExcelFile

from io import StringIO
import requests
 
from sklearn.preprocessing import MinMaxScaler
from keras_anomaly_detection.library.plot_utils import visualize_reconstruction_error
from keras_anomaly_detection.library.convolutional import Conv1DAutoEncoder

import numpy as np
import matplotlib.pyplot as plt

from sklearn.metrics import (confusion_matrix, precision_recall_curve, auc,
                             roc_curve, recall_score, classification_report, f1_score,
                             precision_recall_fscore_support)
from random import random

import seaborn as sns

import tensorflow as tf
from tensorflow import keras
from tensorflow.keras.utils import to_categorical
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Conv2D, MaxPooling2D, Flatten, Dense, Dropout
from tensorflow.keras.losses import categorical_crossentropy
from tensorflow.keras.optimizers import Adam
from tensorflow.keras.preprocessing.image import ImageDataGenerator

import json
import sys
import warnings

if not sys.warnoptions:
    warnings.simplefilter("ignore")
Using TensorFlow backend.

2. Reading the time series data

In order to read the data it can be done by two different methods

Option 1: Read the data directly from BigQuery.

In order to read the data from BigQuery one needs the JSON file containing the authentication key in order to be able to excecute queries involving said data. In this case Im gonna use the file "bigquery-test-228612-bbb8922fe944.json" located at my documents.

  1. We need to define the query. The SELECT parameter goes the data wanted from de table, In this case we want all data contained in the BigQuery table so we use * . The FROM parameter specifies the BigQuery table that holds the data.
In [2]:
query = """
SELECT *
FROM `bigquery-test-228612.ECGData.Test`
LIMIT 1000
"""
  1. We excecute the query and save it into a dataframe, specifying the path to the key file and the project ID that holds the data in the cloud
In [3]:
df = pd.read_gbq(query,
                     project_id='bigquery-test-228612',
                     private_key='C:/Users/DELL/Documents/bigquery-test-228612-9d37266d3392.json',
                     dialect='standard')
  1. We perform some manipulation of the data.
In [4]:
df=df.iloc[1:len(df.index),:]
Col=np.linspace(1,len(df.columns),len(df.columns))
df.columns=Col
df.head()
Out[4]:
1.0 2.0 3.0 4.0 5.0 6.0 7.0 8.0 9.0 10.0 ... 201.0 202.0 203.0 204.0 205.0 206.0 207.0 208.0 209.0 210.0
1 2.10 2.13 2.19 2.28 2.44 2.62 2.80 3.04 3.36 3.69 ... 2.99 2.74 2.61 2.47 2.35 2.26 2.20 2.15 2.10 2.08
2 2.06 2.05 2.06 2.07 2.08 2.13 2.22 2.37 2.53 2.71 ... 3.62 3.38 3.17 2.92 2.64 2.45 2.33 2.23 2.13 2.08
3 2.05 2.05 2.03 2.02 2.03 2.04 2.08 2.14 2.28 2.44 ... 3.71 3.33 3.01 2.74 2.55 2.51 2.30 2.17 2.10 2.08
4 2.07 2.04 2.03 2.05 2.05 2.04 2.05 2.09 2.14 2.22 ... 3.27 3.02 2.74 2.54 2.39 2.27 2.17 2.13 2.11 2.07
5 2.06 2.07 2.07 2.08 2.08 2.12 2.18 2.28 2.43 2.66 ... 2.31 2.22 2.16 2.12 2.08 2.08 2.08 2.07 2.06 2.09

5 rows × 210 columns

Option 2: Read the data directly from the github source

Then, we read the ECG data

In [5]:
url='https://raw.githubusercontent.com/chen0040/keras-anomaly-detection/master/demo/ecg_demo/data/ecg_discord_test.csv'
s=requests.get(url).text
df=pd.read_csv(StringIO(s),header=None)
df.head()
Out[5]:
0 1 2 3 4 5 6 7 8 9 ... 200 201 202 203 204 205 206 207 208 209
0 2.10 2.13 2.19 2.28 2.44 2.62 2.80 3.04 3.36 3.69 ... 2.99 2.74 2.61 2.47 2.35 2.26 2.20 2.15 2.10 2.08
1 2.06 2.05 2.06 2.07 2.08 2.13 2.22 2.37 2.53 2.71 ... 3.62 3.38 3.17 2.92 2.64 2.45 2.33 2.23 2.13 2.08
2 2.05 2.05 2.03 2.02 2.03 2.04 2.08 2.14 2.28 2.44 ... 3.71 3.33 3.01 2.74 2.55 2.51 2.30 2.17 2.10 2.08
3 2.07 2.04 2.03 2.05 2.05 2.04 2.05 2.09 2.14 2.22 ... 3.27 3.02 2.74 2.54 2.39 2.27 2.17 2.13 2.11 2.07
4 2.06 2.07 2.07 2.08 2.08 2.12 2.18 2.28 2.43 2.66 ... 2.31 2.22 2.16 2.12 2.08 2.08 2.08 2.07 2.06 2.09

5 rows × 210 columns

3. EDA and Visualization of the data

Now we perform an EDA over the data

In [6]:
Tdf=df.T
Tdf.describe()
Out[6]:
0 1 2 3 4 5 6 7 8 9 ... 13 14 15 16 17 18 19 20 21 22
count 210.000000 210.000000 210.000000 210.000000 210.000000 210.000000 210.000000 210.000000 210.000000 210.000000 ... 210.000000 210.000000 210.000000 210.000000 210.000000 210.000000 210.000000 210.000000 210.000000 210.000000
mean 5.197714 5.142667 5.091667 5.112571 5.073333 5.052095 5.189333 5.101000 5.157286 5.114429 ... 5.115476 5.139381 5.046619 5.118905 5.158810 5.146905 5.202619 5.040143 5.324238 5.132143
std 1.134086 1.141165 1.124836 1.208544 1.240130 1.219582 1.143697 1.180555 1.190994 1.199702 ... 1.198160 1.211069 1.163220 1.213130 1.176845 1.203633 1.220693 0.867790 1.233114 1.178374
min 2.080000 2.050000 2.020000 2.030000 2.060000 2.080000 2.090000 2.080000 2.090000 1.990000 ... 2.000000 2.040000 2.080000 2.040000 2.070000 2.020000 2.040000 3.660000 2.240000 2.060000
25% 5.010000 4.952500 4.970000 4.990000 4.972500 5.000000 5.032500 4.980000 5.000000 4.930000 ... 4.930000 4.940000 4.980000 4.970000 4.970000 4.962500 4.970000 4.502500 5.070000 5.020000
50% 5.140000 5.090000 5.110000 5.195000 5.170000 5.170000 5.220000 5.130000 5.120000 5.090000 ... 5.090000 5.090000 5.120000 5.170000 5.090000 5.115000 5.155000 5.040000 5.240000 5.100000
75% 5.807500 5.775000 5.697500 5.810000 5.825000 5.785000 5.840000 5.712500 5.825000 5.790000 ... 5.922500 5.932500 5.605000 5.735000 5.860000 5.910000 6.035000 5.207500 6.065000 5.760000
max 7.080000 7.050000 6.820000 6.930000 6.890000 6.840000 7.010000 6.950000 7.100000 7.040000 ... 6.870000 6.980000 6.940000 7.070000 7.090000 6.980000 6.970000 7.640000 7.410000 7.030000

8 rows × 23 columns

In [7]:
# The number and indexes of the series you want to plot
Tdf=df.iloc[15:17]
Tdf=Tdf.T
Tdf.plot()
Out[7]:
<matplotlib.axes._subplots.AxesSubplot at 0x18999f1c208>
In [8]:
sns.pairplot(df.T)
Out[8]:
<seaborn.axisgrid.PairGrid at 0x18999ff4b70>
In [9]:
corr = (np.transpose(df)).corr()

f, ax = plt.subplots(figsize=(14, 7))
hm = sns.heatmap(corr, annot=True, ax=ax, cmap="coolwarm",fmt='.2f',
                 linewidths=.05)
f.subplots_adjust(top=0.93)
t= f.suptitle('Correlation Heatmap', fontsize=14)
In [10]:
df['ID'] = pd.Series(np.linspace(1,len(df.index),len(df.index)), index=df.index)
Data = pd.melt(df.iloc[0:10,:],id_vars="ID",var_name="Series",value_name='Value') # Boxplot of the first 10 series
plt.figure(figsize=(10, 10))
sns.boxplot(x="ID", y="Value", data=Data)
Out[10]:
<matplotlib.axes._subplots.AxesSubplot at 0x189aed51978>

4. Generation of New Data for Training and Validation

After the EDA we add some artificial outliers and inliers. Given the distribution of the data and its maximum and minimum values the new data was created as a continuous random number between 2 and 7

In [11]:
# Generate normal series
NormalSeriesNew=7   #How many new normal data you want 

for i in range (NormalSeriesNew):
    ls=[]
    for j in range(len(df.columns)):
        value = random()
        ls.append(2+(value*5))
    numEl = len(ls)
    newRow = pd.DataFrame(np.array(ls).reshape(1,numEl), columns = list(df.columns))
    df = df.append(newRow, ignore_index=True)
    
# Generate abnormal series  
AbnormalSeriesNew=5 #How many new normal data you want 
for i in range (AbnormalSeriesNew):
    ls=[]
    for j in range(len(df.columns)):
        value = random()
        ls.append((2+(value*5))*5)
    numEl = len(ls)
    newRow = pd.DataFrame(np.array(ls).reshape(1,numEl), columns = list(df.columns))
    df = df.append(newRow, ignore_index=True)
df
Out[11]:
0 1 2 3 4 5 6 7 8 9 ... 201 202 203 204 205 206 207 208 209 ID
0 2.100000 2.130000 2.190000 2.280000 2.440000 2.620000 2.800000 3.040000 3.360000 3.690000 ... 2.740000 2.610000 2.470000 2.350000 2.260000 2.200000 2.150000 2.100000 2.080000 1.000000
1 2.060000 2.050000 2.060000 2.070000 2.080000 2.130000 2.220000 2.370000 2.530000 2.710000 ... 3.380000 3.170000 2.920000 2.640000 2.450000 2.330000 2.230000 2.130000 2.080000 2.000000
2 2.050000 2.050000 2.030000 2.020000 2.030000 2.040000 2.080000 2.140000 2.280000 2.440000 ... 3.330000 3.010000 2.740000 2.550000 2.510000 2.300000 2.170000 2.100000 2.080000 3.000000
3 2.070000 2.040000 2.030000 2.050000 2.050000 2.040000 2.050000 2.090000 2.140000 2.220000 ... 3.020000 2.740000 2.540000 2.390000 2.270000 2.170000 2.130000 2.110000 2.070000 4.000000
4 2.060000 2.070000 2.070000 2.080000 2.080000 2.120000 2.180000 2.280000 2.430000 2.660000 ... 2.220000 2.160000 2.120000 2.080000 2.080000 2.080000 2.070000 2.060000 2.090000 5.000000
5 2.100000 2.180000 2.280000 2.460000 2.690000 2.980000 3.300000 3.640000 3.980000 4.280000 ... 2.110000 2.130000 2.210000 2.350000 2.530000 2.750000 3.030000 3.400000 3.760000 6.000000
6 4.070000 4.380000 4.690000 4.930000 5.130000 5.340000 5.550000 5.700000 5.780000 5.840000 ... 4.760000 5.060000 5.310000 5.520000 5.670000 5.780000 5.840000 5.900000 5.940000 7.000000
7 6.000000 6.040000 6.080000 6.100000 6.140000 6.160000 6.210000 6.240000 6.280000 6.330000 ... 4.690000 4.920000 5.120000 5.310000 5.370000 5.640000 5.730000 5.800000 5.870000 8.000000
8 5.930000 5.970000 6.000000 6.060000 6.100000 6.150000 6.180000 6.240000 6.280000 6.330000 ... 2.830000 3.110000 3.450000 3.790000 4.090000 4.410000 4.690000 4.950000 5.170000 9.000000
9 5.360000 5.530000 5.650000 5.740000 5.800000 5.860000 5.930000 5.970000 6.000000 6.050000 ... 2.770000 3.120000 3.480000 3.780000 4.110000 4.450000 4.720000 4.950000 5.190000 10.000000
10 5.260000 5.570000 5.660000 5.740000 5.800000 5.840000 5.870000 5.910000 5.960000 6.000000 ... 3.500000 3.810000 4.150000 4.460000 4.740000 4.970000 5.190000 5.400000 5.580000 11.000000
11 5.700000 5.770000 5.840000 5.880000 5.910000 5.940000 5.980000 6.020000 6.040000 6.050000 ... 5.640000 5.740000 5.800000 5.840000 5.890000 5.940000 5.980000 6.000000 6.030000 12.000000
12 6.090000 6.110000 6.140000 6.170000 6.200000 6.210000 6.280000 6.310000 6.370000 6.420000 ... 5.990000 6.000000 6.040000 6.080000 6.090000 6.140000 6.160000 6.200000 6.240000 13.000000
13 6.270000 6.290000 6.340000 6.380000 6.420000 6.450000 6.490000 6.550000 6.580000 6.600000 ... 6.240000 6.270000 6.330000 6.370000 6.410000 6.450000 6.510000 6.560000 6.590000 14.000000
14 6.630000 6.680000 6.730000 6.760000 6.800000 6.850000 6.880000 6.910000 6.930000 6.960000 ... 6.490000 6.530000 6.550000 6.580000 6.630000 6.690000 6.720000 6.760000 6.810000 15.000000
15 6.830000 6.880000 6.880000 6.910000 6.940000 6.930000 6.920000 6.910000 6.890000 6.860000 ... 6.320000 6.340000 6.380000 6.420000 6.460000 6.480000 6.520000 6.580000 6.620000 16.000000
16 6.660000 6.690000 6.740000 6.800000 6.810000 6.860000 6.910000 6.950000 6.970000 7.000000 ... 6.030000 6.080000 6.120000 6.150000 6.190000 6.250000 6.300000 6.330000 6.370000 17.000000
17 6.420000 6.480000 6.510000 6.560000 6.610000 6.630000 6.710000 6.750000 6.800000 6.850000 ... 6.060000 6.090000 6.130000 6.170000 6.190000 6.270000 6.300000 6.360000 6.410000 18.000000
18 6.460000 6.490000 6.550000 6.610000 6.650000 6.680000 6.720000 6.780000 6.820000 6.860000 ... 6.310000 6.360000 6.410000 6.460000 6.490000 6.530000 6.600000 6.650000 6.670000 19.000000
19 6.710000 6.770000 6.820000 6.840000 6.870000 6.910000 6.940000 6.960000 6.950000 6.960000 ... 6.710000 6.750000 6.770000 6.800000 6.840000 6.860000 6.890000 6.890000 6.900000 20.000000
20 6.900000 6.890000 6.860000 6.820000 6.780000 6.730000 6.640000 6.570000 6.500000 6.410000 ... 5.030000 5.030000 5.020000 5.040000 5.040000 5.030000 5.030000 5.050000 5.040000 21.000000
21 5.040000 5.040000 5.040000 5.050000 5.050000 5.050000 5.040000 5.050000 5.050000 5.040000 ... 5.270000 5.270000 5.260000 5.230000 5.220000 5.210000 5.200000 5.180000 5.150000 22.000000
22 5.160000 5.150000 5.140000 5.140000 5.160000 5.170000 5.150000 5.140000 5.130000 5.130000 ... 5.080000 5.070000 5.060000 5.050000 5.050000 5.060000 5.040000 5.030000 5.030000 23.000000
23 5.804213 2.012050 4.089851 5.431032 6.294707 6.407326 2.272181 5.738539 2.899624 3.436274 ... 2.038050 6.626270 5.348071 2.792930 2.370228 6.734223 4.919572 6.729869 3.598187 2.169168
24 2.793436 4.237529 2.066581 3.524060 6.868945 4.496093 2.094637 4.609838 5.337058 6.625311 ... 2.514861 3.953349 4.595274 5.250058 3.374134 4.754918 2.992002 4.859303 6.826402 2.881342
25 6.119625 4.436260 6.691462 5.699218 6.914172 5.108834 4.603016 2.126197 5.644423 3.236014 ... 3.167436 5.363156 6.473694 3.323180 2.050288 5.636039 6.248945 4.469255 6.342950 5.994035
26 3.187350 6.840972 6.636128 6.070335 6.954283 3.140779 4.752588 4.575225 3.522036 6.971140 ... 4.063359 6.878479 5.414163 5.104304 6.391342 2.860069 6.979819 4.070653 3.272461 4.523576
27 3.967523 4.855274 3.530429 4.614740 4.714547 2.256423 4.258872 2.274494 5.947641 2.325212 ... 3.405488 3.248402 4.146609 3.067796 3.067548 5.797208 4.563009 2.314628 3.690112 3.939399
28 6.173307 3.222203 6.253150 2.699540 5.701320 3.824927 4.376028 4.529612 3.610153 4.109254 ... 4.696746 5.086205 4.076932 6.539127 3.090105 2.169890 4.439752 5.504714 6.547337 5.595228
29 5.232097 4.503427 4.320138 2.304386 3.766598 3.217730 2.316847 5.968223 2.787674 3.821709 ... 2.032560 6.765822 6.151503 6.962642 5.207201 6.866021 3.756540 4.406737 5.162345 6.618891
30 16.394026 11.374236 34.455634 17.893248 19.166615 27.709693 24.085360 30.444181 12.184815 22.388763 ... 14.767659 27.372737 18.749360 12.964196 23.856454 22.090446 30.104660 24.297814 32.948479 34.563216
31 16.086020 14.884804 13.000825 25.693291 27.040104 28.763945 32.981025 21.762465 22.621909 18.163670 ... 30.485056 12.178837 10.348366 20.079210 34.337486 29.155381 23.198755 11.739537 14.675381 18.208239
32 19.981689 13.176698 28.718324 34.936490 26.457105 24.869250 31.930620 29.266991 32.284550 33.171853 ... 28.743722 10.996253 23.875236 15.033072 17.866121 11.747734 25.506936 27.198151 28.110930 17.992182
33 23.958417 16.809918 19.652079 15.787753 13.275052 26.071565 17.449696 19.587028 22.760577 20.754690 ... 11.169368 18.209375 10.232598 23.392227 22.348390 26.948959 18.599806 15.105130 21.064672 10.670335
34 23.014737 32.061023 16.100497 29.114415 26.308745 24.702104 18.504777 12.736899 10.010763 23.421689 ... 34.112520 32.036562 12.386043 17.872052 22.354915 22.917453 21.119430 28.489327 12.198773 11.598936

35 rows × 211 columns

With the complete data we create the labels for the outliers (1) and inliers (0)

In [12]:
labels=[]
Tdf= df.T
for i in range(len(df.index)):
    if Tdf[i].mean()<10:
        labels.append(0)
    else:
        labels.append(1)

5. Construcction of the NN for Anomaly Detection

Now with the data and labels we construct the NN model for the anomaly detection

In [13]:
from keras.callbacks import TensorBoard
from keras.layers import Conv1D, GlobalMaxPool1D, Dense, Flatten
from keras.models import Sequential
from keras.callbacks import ModelCheckpoint
from numpy.random import seed

tensorboard = TensorBoard(log_dir='./logs', histogram_freq=0,write_graph=True, write_images=False)
df_np_data = df.values
scaler = MinMaxScaler()
df_np_data = scaler.fit_transform(df_np_data)

model = Sequential()
model.add(Conv1D(filters=256, kernel_size=5, padding='same', activation='relu', input_shape=(df_np_data.shape[1], 1)))
model.add(GlobalMaxPool1D())
model.add(Dense(units=df_np_data.shape[1], activation='linear'))
model.compile(optimizer='adam', loss='mean_squared_error', metrics=['mean_absolute_error'])

Xin=np.expand_dims(df_np_data, axis=2)
Yin=df_np_data

seed(999)
model.fit(x=Xin, y=Yin,batch_size=8, epochs=100,verbose=2 ,validation_split=0.1, callbacks=[tensorboard])
WARNING:tensorflow:From C:\Users\DELL\AppData\Local\Continuum\anaconda3\lib\site-packages\keras\backend\tensorflow_backend.py:1213: calling reduce_max (from tensorflow.python.ops.math_ops) with keep_dims is deprecated and will be removed in a future version.
Instructions for updating:
keep_dims is deprecated, use keepdims instead
WARNING:tensorflow:From C:\Users\DELL\AppData\Local\Continuum\anaconda3\lib\site-packages\keras\backend\tensorflow_backend.py:1213: calling reduce_max (from tensorflow.python.ops.math_ops) with keep_dims is deprecated and will be removed in a future version.
Instructions for updating:
keep_dims is deprecated, use keepdims instead
WARNING:tensorflow:From C:\Users\DELL\AppData\Local\Continuum\anaconda3\lib\site-packages\keras\backend\tensorflow_backend.py:1349: calling reduce_mean (from tensorflow.python.ops.math_ops) with keep_dims is deprecated and will be removed in a future version.
Instructions for updating:
keep_dims is deprecated, use keepdims instead
WARNING:tensorflow:From C:\Users\DELL\AppData\Local\Continuum\anaconda3\lib\site-packages\keras\backend\tensorflow_backend.py:1349: calling reduce_mean (from tensorflow.python.ops.math_ops) with keep_dims is deprecated and will be removed in a future version.
Instructions for updating:
keep_dims is deprecated, use keepdims instead
Train on 31 samples, validate on 4 samples
Epoch 1/100
 - 0s - loss: 0.0290 - mean_absolute_error: 0.1094 - val_loss: 0.4661 - val_mean_absolute_error: 0.6345
Epoch 2/100
 - 0s - loss: 0.0238 - mean_absolute_error: 0.0931 - val_loss: 0.3948 - val_mean_absolute_error: 0.5752
Epoch 3/100
 - 0s - loss: 0.0190 - mean_absolute_error: 0.0766 - val_loss: 0.3234 - val_mean_absolute_error: 0.5092
Epoch 4/100
 - 0s - loss: 0.0144 - mean_absolute_error: 0.0620 - val_loss: 0.2573 - val_mean_absolute_error: 0.4414
Epoch 5/100
 - 0s - loss: 0.0121 - mean_absolute_error: 0.0558 - val_loss: 0.2001 - val_mean_absolute_error: 0.3783
Epoch 6/100
 - 0s - loss: 0.0104 - mean_absolute_error: 0.0554 - val_loss: 0.1642 - val_mean_absolute_error: 0.3366
Epoch 7/100
 - 0s - loss: 0.0092 - mean_absolute_error: 0.0572 - val_loss: 0.1493 - val_mean_absolute_error: 0.3187
Epoch 8/100
 - 0s - loss: 0.0084 - mean_absolute_error: 0.0562 - val_loss: 0.1463 - val_mean_absolute_error: 0.3152
Epoch 9/100
 - 0s - loss: 0.0077 - mean_absolute_error: 0.0533 - val_loss: 0.1522 - val_mean_absolute_error: 0.3224
Epoch 10/100
 - 0s - loss: 0.0070 - mean_absolute_error: 0.0501 - val_loss: 0.1481 - val_mean_absolute_error: 0.3176
Epoch 11/100
 - 0s - loss: 0.0067 - mean_absolute_error: 0.0497 - val_loss: 0.1369 - val_mean_absolute_error: 0.3040
Epoch 12/100
 - 0s - loss: 0.0060 - mean_absolute_error: 0.0480 - val_loss: 0.1317 - val_mean_absolute_error: 0.2977
Epoch 13/100
 - 0s - loss: 0.0055 - mean_absolute_error: 0.0466 - val_loss: 0.1209 - val_mean_absolute_error: 0.2846
Epoch 14/100
 - 0s - loss: 0.0051 - mean_absolute_error: 0.0461 - val_loss: 0.1092 - val_mean_absolute_error: 0.2707
Epoch 15/100
 - 0s - loss: 0.0048 - mean_absolute_error: 0.0460 - val_loss: 0.1006 - val_mean_absolute_error: 0.2606
Epoch 16/100
 - 0s - loss: 0.0042 - mean_absolute_error: 0.0444 - val_loss: 0.0981 - val_mean_absolute_error: 0.2579
Epoch 17/100
 - 0s - loss: 0.0040 - mean_absolute_error: 0.0434 - val_loss: 0.0950 - val_mean_absolute_error: 0.2543
Epoch 18/100
 - 0s - loss: 0.0035 - mean_absolute_error: 0.0423 - val_loss: 0.0935 - val_mean_absolute_error: 0.2525
Epoch 19/100
 - 0s - loss: 0.0033 - mean_absolute_error: 0.0414 - val_loss: 0.0928 - val_mean_absolute_error: 0.2519
Epoch 20/100
 - 0s - loss: 0.0031 - mean_absolute_error: 0.0407 - val_loss: 0.0917 - val_mean_absolute_error: 0.2508
Epoch 21/100
 - 0s - loss: 0.0029 - mean_absolute_error: 0.0400 - val_loss: 0.0922 - val_mean_absolute_error: 0.2513
Epoch 22/100
 - 0s - loss: 0.0026 - mean_absolute_error: 0.0391 - val_loss: 0.0930 - val_mean_absolute_error: 0.2523
Epoch 23/100
 - 0s - loss: 0.0025 - mean_absolute_error: 0.0385 - val_loss: 0.0949 - val_mean_absolute_error: 0.2542
Epoch 24/100
 - 0s - loss: 0.0024 - mean_absolute_error: 0.0378 - val_loss: 0.0967 - val_mean_absolute_error: 0.2560
Epoch 25/100
 - 0s - loss: 0.0023 - mean_absolute_error: 0.0374 - val_loss: 0.1003 - val_mean_absolute_error: 0.2600
Epoch 26/100
 - 0s - loss: 0.0022 - mean_absolute_error: 0.0368 - val_loss: 0.1033 - val_mean_absolute_error: 0.2632
Epoch 27/100
 - 0s - loss: 0.0021 - mean_absolute_error: 0.0364 - val_loss: 0.1066 - val_mean_absolute_error: 0.2670
Epoch 28/100
 - 0s - loss: 0.0021 - mean_absolute_error: 0.0358 - val_loss: 0.1092 - val_mean_absolute_error: 0.2698
Epoch 29/100
 - 0s - loss: 0.0020 - mean_absolute_error: 0.0353 - val_loss: 0.1107 - val_mean_absolute_error: 0.2715
Epoch 30/100
 - 0s - loss: 0.0020 - mean_absolute_error: 0.0349 - val_loss: 0.1118 - val_mean_absolute_error: 0.2727
Epoch 31/100
 - 0s - loss: 0.0020 - mean_absolute_error: 0.0347 - val_loss: 0.1128 - val_mean_absolute_error: 0.2739
Epoch 32/100
 - 0s - loss: 0.0019 - mean_absolute_error: 0.0344 - val_loss: 0.1137 - val_mean_absolute_error: 0.2750
Epoch 33/100
 - 0s - loss: 0.0019 - mean_absolute_error: 0.0340 - val_loss: 0.1144 - val_mean_absolute_error: 0.2759
Epoch 34/100
 - 0s - loss: 0.0019 - mean_absolute_error: 0.0337 - val_loss: 0.1152 - val_mean_absolute_error: 0.2768
Epoch 35/100
 - 0s - loss: 0.0019 - mean_absolute_error: 0.0335 - val_loss: 0.1159 - val_mean_absolute_error: 0.2778
Epoch 36/100
 - 0s - loss: 0.0019 - mean_absolute_error: 0.0334 - val_loss: 0.1161 - val_mean_absolute_error: 0.2780
Epoch 37/100
 - 0s - loss: 0.0018 - mean_absolute_error: 0.0332 - val_loss: 0.1164 - val_mean_absolute_error: 0.2785
Epoch 38/100
 - 0s - loss: 0.0018 - mean_absolute_error: 0.0329 - val_loss: 0.1167 - val_mean_absolute_error: 0.2789
Epoch 39/100
 - 0s - loss: 0.0018 - mean_absolute_error: 0.0326 - val_loss: 0.1166 - val_mean_absolute_error: 0.2788
Epoch 40/100
 - 0s - loss: 0.0018 - mean_absolute_error: 0.0325 - val_loss: 0.1164 - val_mean_absolute_error: 0.2786
Epoch 41/100
 - 0s - loss: 0.0018 - mean_absolute_error: 0.0324 - val_loss: 0.1165 - val_mean_absolute_error: 0.2787
Epoch 42/100
 - 0s - loss: 0.0018 - mean_absolute_error: 0.0321 - val_loss: 0.1164 - val_mean_absolute_error: 0.2787
Epoch 43/100
 - 0s - loss: 0.0017 - mean_absolute_error: 0.0320 - val_loss: 0.1165 - val_mean_absolute_error: 0.2788
Epoch 44/100
 - 0s - loss: 0.0017 - mean_absolute_error: 0.0318 - val_loss: 0.1164 - val_mean_absolute_error: 0.2787
Epoch 45/100
 - 0s - loss: 0.0017 - mean_absolute_error: 0.0317 - val_loss: 0.1163 - val_mean_absolute_error: 0.2787
Epoch 46/100
 - 0s - loss: 0.0017 - mean_absolute_error: 0.0316 - val_loss: 0.1165 - val_mean_absolute_error: 0.2790
Epoch 47/100
 - 0s - loss: 0.0017 - mean_absolute_error: 0.0313 - val_loss: 0.1165 - val_mean_absolute_error: 0.2790
Epoch 48/100
 - 0s - loss: 0.0017 - mean_absolute_error: 0.0312 - val_loss: 0.1163 - val_mean_absolute_error: 0.2788
Epoch 49/100
 - 0s - loss: 0.0016 - mean_absolute_error: 0.0311 - val_loss: 0.1163 - val_mean_absolute_error: 0.2789
Epoch 50/100
 - 0s - loss: 0.0016 - mean_absolute_error: 0.0309 - val_loss: 0.1163 - val_mean_absolute_error: 0.2790
Epoch 51/100
 - 0s - loss: 0.0016 - mean_absolute_error: 0.0307 - val_loss: 0.1168 - val_mean_absolute_error: 0.2796
Epoch 52/100
 - 0s - loss: 0.0016 - mean_absolute_error: 0.0304 - val_loss: 0.1171 - val_mean_absolute_error: 0.2800
Epoch 53/100
 - 0s - loss: 0.0016 - mean_absolute_error: 0.0302 - val_loss: 0.1166 - val_mean_absolute_error: 0.2795
Epoch 54/100
 - 0s - loss: 0.0016 - mean_absolute_error: 0.0301 - val_loss: 0.1164 - val_mean_absolute_error: 0.2793
Epoch 55/100
 - 0s - loss: 0.0016 - mean_absolute_error: 0.0300 - val_loss: 0.1165 - val_mean_absolute_error: 0.2793
Epoch 56/100
 - 0s - loss: 0.0015 - mean_absolute_error: 0.0298 - val_loss: 0.1165 - val_mean_absolute_error: 0.2794
Epoch 57/100
 - 0s - loss: 0.0015 - mean_absolute_error: 0.0297 - val_loss: 0.1160 - val_mean_absolute_error: 0.2788
Epoch 58/100
 - 0s - loss: 0.0015 - mean_absolute_error: 0.0295 - val_loss: 0.1162 - val_mean_absolute_error: 0.2791
Epoch 59/100
 - 0s - loss: 0.0015 - mean_absolute_error: 0.0293 - val_loss: 0.1166 - val_mean_absolute_error: 0.2795
Epoch 60/100
 - 0s - loss: 0.0015 - mean_absolute_error: 0.0291 - val_loss: 0.1167 - val_mean_absolute_error: 0.2796
Epoch 61/100
 - 0s - loss: 0.0015 - mean_absolute_error: 0.0290 - val_loss: 0.1164 - val_mean_absolute_error: 0.2794
Epoch 62/100
 - 0s - loss: 0.0015 - mean_absolute_error: 0.0289 - val_loss: 0.1163 - val_mean_absolute_error: 0.2793
Epoch 63/100
 - 0s - loss: 0.0014 - mean_absolute_error: 0.0286 - val_loss: 0.1169 - val_mean_absolute_error: 0.2800
Epoch 64/100
 - 0s - loss: 0.0014 - mean_absolute_error: 0.0283 - val_loss: 0.1173 - val_mean_absolute_error: 0.2805
Epoch 65/100
 - 0s - loss: 0.0014 - mean_absolute_error: 0.0282 - val_loss: 0.1168 - val_mean_absolute_error: 0.2799
Epoch 66/100
 - 0s - loss: 0.0014 - mean_absolute_error: 0.0281 - val_loss: 0.1169 - val_mean_absolute_error: 0.2802
Epoch 67/100
 - 0s - loss: 0.0014 - mean_absolute_error: 0.0280 - val_loss: 0.1170 - val_mean_absolute_error: 0.2803
Epoch 68/100
 - 0s - loss: 0.0014 - mean_absolute_error: 0.0278 - val_loss: 0.1168 - val_mean_absolute_error: 0.2801
Epoch 69/100
 - 0s - loss: 0.0014 - mean_absolute_error: 0.0277 - val_loss: 0.1168 - val_mean_absolute_error: 0.2801
Epoch 70/100
 - 0s - loss: 0.0014 - mean_absolute_error: 0.0275 - val_loss: 0.1171 - val_mean_absolute_error: 0.2804
Epoch 71/100
 - 0s - loss: 0.0013 - mean_absolute_error: 0.0273 - val_loss: 0.1175 - val_mean_absolute_error: 0.2808
Epoch 72/100
 - 0s - loss: 0.0013 - mean_absolute_error: 0.0272 - val_loss: 0.1174 - val_mean_absolute_error: 0.2808
Epoch 73/100
 - 0s - loss: 0.0013 - mean_absolute_error: 0.0271 - val_loss: 0.1172 - val_mean_absolute_error: 0.2806
Epoch 74/100
 - 0s - loss: 0.0013 - mean_absolute_error: 0.0269 - val_loss: 0.1175 - val_mean_absolute_error: 0.2811
Epoch 75/100
 - 0s - loss: 0.0013 - mean_absolute_error: 0.0267 - val_loss: 0.1175 - val_mean_absolute_error: 0.2810
Epoch 76/100
 - 0s - loss: 0.0013 - mean_absolute_error: 0.0266 - val_loss: 0.1179 - val_mean_absolute_error: 0.2815
Epoch 77/100
 - 0s - loss: 0.0013 - mean_absolute_error: 0.0265 - val_loss: 0.1180 - val_mean_absolute_error: 0.2816
Epoch 78/100
 - 0s - loss: 0.0013 - mean_absolute_error: 0.0264 - val_loss: 0.1178 - val_mean_absolute_error: 0.2814
Epoch 79/100
 - 0s - loss: 0.0013 - mean_absolute_error: 0.0262 - val_loss: 0.1177 - val_mean_absolute_error: 0.2813
Epoch 80/100
 - 0s - loss: 0.0013 - mean_absolute_error: 0.0262 - val_loss: 0.1179 - val_mean_absolute_error: 0.2815
Epoch 81/100
 - 0s - loss: 0.0012 - mean_absolute_error: 0.0260 - val_loss: 0.1187 - val_mean_absolute_error: 0.2825
Epoch 82/100
 - 0s - loss: 0.0012 - mean_absolute_error: 0.0259 - val_loss: 0.1188 - val_mean_absolute_error: 0.2827
Epoch 83/100
 - 0s - loss: 0.0012 - mean_absolute_error: 0.0259 - val_loss: 0.1180 - val_mean_absolute_error: 0.2817
Epoch 84/100
 - 0s - loss: 0.0012 - mean_absolute_error: 0.0258 - val_loss: 0.1182 - val_mean_absolute_error: 0.2818
Epoch 85/100
 - 0s - loss: 0.0012 - mean_absolute_error: 0.0256 - val_loss: 0.1186 - val_mean_absolute_error: 0.2824
Epoch 86/100
 - 0s - loss: 0.0012 - mean_absolute_error: 0.0254 - val_loss: 0.1193 - val_mean_absolute_error: 0.2833
Epoch 87/100
 - 0s - loss: 0.0012 - mean_absolute_error: 0.0253 - val_loss: 0.1192 - val_mean_absolute_error: 0.2832
Epoch 88/100
 - 0s - loss: 0.0012 - mean_absolute_error: 0.0255 - val_loss: 0.1189 - val_mean_absolute_error: 0.2828
Epoch 89/100
 - 0s - loss: 0.0012 - mean_absolute_error: 0.0253 - val_loss: 0.1186 - val_mean_absolute_error: 0.2825
Epoch 90/100
 - 0s - loss: 0.0012 - mean_absolute_error: 0.0250 - val_loss: 0.1192 - val_mean_absolute_error: 0.2832
Epoch 91/100
 - 0s - loss: 0.0012 - mean_absolute_error: 0.0249 - val_loss: 0.1199 - val_mean_absolute_error: 0.2841
Epoch 92/100
 - 0s - loss: 0.0012 - mean_absolute_error: 0.0250 - val_loss: 0.1197 - val_mean_absolute_error: 0.2839
Epoch 93/100
 - 0s - loss: 0.0012 - mean_absolute_error: 0.0249 - val_loss: 0.1194 - val_mean_absolute_error: 0.2834
Epoch 94/100
 - 0s - loss: 0.0012 - mean_absolute_error: 0.0247 - val_loss: 0.1194 - val_mean_absolute_error: 0.2835
Epoch 95/100
 - 0s - loss: 0.0012 - mean_absolute_error: 0.0247 - val_loss: 0.1194 - val_mean_absolute_error: 0.2834
Epoch 96/100
 - 0s - loss: 0.0012 - mean_absolute_error: 0.0248 - val_loss: 0.1197 - val_mean_absolute_error: 0.2838
Epoch 97/100
 - 0s - loss: 0.0011 - mean_absolute_error: 0.0245 - val_loss: 0.1203 - val_mean_absolute_error: 0.2846
Epoch 98/100
 - 0s - loss: 0.0011 - mean_absolute_error: 0.0245 - val_loss: 0.1203 - val_mean_absolute_error: 0.2847
Epoch 99/100
 - 0s - loss: 0.0011 - mean_absolute_error: 0.0243 - val_loss: 0.1201 - val_mean_absolute_error: 0.2843
Epoch 100/100
 - 0s - loss: 0.0011 - mean_absolute_error: 0.0245 - val_loss: 0.1197 - val_mean_absolute_error: 0.2838
Out[13]:
<keras.callbacks.History at 0x189b6c2dfd0>

In order to view the results in tensorboard, open the console and type: tensorboard --logdir C:\Users\DELL\Documents\logs --host=127.0.0.1 and press enter

After pressing enter the console will show a direction. Open your browser and paste the direction. This direction will send you to the tensorboard panel

The argument "C:\Users\DELL\Documents\logs" refers to your log directory

It should show you something like this

In [14]:
from IPython.display import Image

i = Image(filename='C:/Users/DELL/Pictures/TensorBoard/Conv1D/1.PNG')
i
Out[14]:
In [15]:
i = Image(filename='C:/Users/DELL/Pictures/TensorBoard/Conv1D/2.PNG')
i
Out[15]:
In [16]:
i = i = Image(filename='C:/Users/DELL/Pictures/TensorBoard/Conv1D/3.PNG')
i
Out[16]:

6. Performance and Validation of the NN

Now the anomaly detection

In [17]:
## For each of the observations it predicts wheter or not is an anomaly
estimated_negative_sample_ratio = 0.9
input_timeseries_dataset = np.expand_dims(df_np_data , axis=2)
target_timeseries_dataset = model.predict(x=input_timeseries_dataset)
dist = np.linalg.norm(df_np_data  - target_timeseries_dataset, axis=-1)
scores = dist
scores.sort()
cut_point = int(estimated_negative_sample_ratio * len(scores))
threshold = scores[cut_point]

dist = np.linalg.norm(df_np_data  - target_timeseries_dataset, axis=-1)

Result=zip(dist >= threshold, dist)

reconstruction_error = []
for idx, (is_anomaly, dist) in enumerate(Result):
    print('# ' + str(idx) + ' is ' + ('abnormal' if is_anomaly else 'normal') + ' (dist: ' + str(dist) + ')')
    reconstruction_error.append(dist)

# Visualize the anomalies
visualize_reconstruction_error(reconstruction_error, threshold)
# 0 is normal (dist: 0.32032729372184576)
# 1 is normal (dist: 0.3529996426211889)
# 2 is normal (dist: 0.3846988186362842)
# 3 is normal (dist: 0.4180971749644787)
# 4 is normal (dist: 0.3650565405196149)
# 5 is normal (dist: 0.343529768523278)
# 6 is normal (dist: 0.341283981874237)
# 7 is normal (dist: 0.3894456530945373)
# 8 is normal (dist: 0.38254398371698817)
# 9 is normal (dist: 0.3838508653093652)
# 10 is normal (dist: 0.3787204341889207)
# 11 is normal (dist: 0.3597045786159159)
# 12 is normal (dist: 0.3671465472517718)
# 13 is normal (dist: 0.3720903019724554)
# 14 is normal (dist: 0.4502549927341108)
# 15 is normal (dist: 0.3546673856706506)
# 16 is normal (dist: 0.3276874014641199)
# 17 is normal (dist: 0.3161838808450201)
# 18 is normal (dist: 0.32741943881508107)
# 19 is normal (dist: 0.5080159915958558)
# 20 is normal (dist: 0.49803245777052135)
# 21 is normal (dist: 0.593622563058442)
# 22 is normal (dist: 0.529662889498522)
# 23 is normal (dist: 0.7256768270162959)
# 24 is normal (dist: 0.6957688457287857)
# 25 is normal (dist: 0.7295197984291183)
# 26 is normal (dist: 0.7559778429634926)
# 27 is normal (dist: 0.6664233926488764)
# 28 is normal (dist: 0.712601794265125)
# 29 is normal (dist: 0.7107054067947187)
# 30 is normal (dist: 0.1079821607516247)
# 31 is abnormal (dist: 5.223082986417019)
# 32 is abnormal (dist: 5.228236689567082)
# 33 is abnormal (dist: 4.761477562447666)
# 34 is abnormal (dist: 4.876347428784878)

Now the ROC Curve and the confusion matrix

In [18]:
TClass=pd.DataFrame(labels,columns=["Class"])
TClass['Rec_Error'] = pd.Series(reconstruction_error).values
TClass
Out[18]:
Class Rec_Error
0 0 0.320327
1 0 0.353000
2 0 0.384699
3 0 0.418097
4 0 0.365057
5 0 0.343530
6 0 0.341284
7 0 0.389446
8 0 0.382544
9 0 0.383851
10 0 0.378720
11 0 0.359705
12 0 0.367147
13 0 0.372090
14 0 0.450255
15 0 0.354667
16 0 0.327687
17 0 0.316184
18 0 0.327419
19 0 0.508016
20 0 0.498032
21 0 0.593623
22 0 0.529663
23 0 0.725677
24 0 0.695769
25 0 0.729520
26 0 0.755978
27 0 0.666423
28 0 0.712602
29 0 0.710705
30 1 0.107982
31 1 5.223083
32 1 5.228237
33 1 4.761478
34 1 4.876347
In [19]:
## ROC CURVE
fpr, tpr, thresholds = roc_curve(TClass.Class, TClass.Rec_Error)
roc_auc = auc(fpr, tpr)

plt.title('Receiver Operating Characteristic')
plt.plot(fpr, tpr, label='AUC = %0.4f'% roc_auc)
plt.legend(loc='lower right')
plt.plot([0,1],[0,1],'r--')
plt.xlim([-0.01, 1.01])
plt.ylim([0, 1.001])
plt.ylabel('True Positive Rate')
plt.xlabel('False Positive Rate')
plt.show()
In [20]:
#Quality  metrics and graphs for the model of anomaly detection

precision, recall, th = precision_recall_curve(TClass.Class, TClass.Rec_Error)
plt.plot(recall, precision, 'b', label='Precision-Recall curve')
plt.title('Recall vs Precision')
plt.xlabel('Recall')
plt.ylabel('Precision')
plt.show()


plt.plot(th, precision[1:], 'b', label='Threshold-Precision curve')
plt.title('Precision for different threshold values')
plt.xlabel('Threshold')
plt.ylabel('Precision')
plt.show()


plt.plot(th, recall[1:], 'b', label='Threshold-Recall curve')
plt.title('Recall for different threshold values')
plt.xlabel('Reconstruction error')
plt.ylabel('Recall')
plt.show()
In [21]:
## CONFUSSION MATRIX
LABELS = ["Normal", "Outlier"]

y_pred = [1 if e >= threshold else 0 for e in TClass.Rec_Error.values]
conf_matrix = confusion_matrix(TClass.Class, y_pred)
plt.figure(figsize=(12, 12))
sns.heatmap(conf_matrix, xticklabels=LABELS, yticklabels=LABELS, annot=True, fmt="d");
plt.title("Confusion matrix")
plt.ylabel('True class')
plt.xlabel('Predicted class')
plt.show()

7. Deployment of the NN to Google Cloud

The model we trained needs to be in keras with this configuration

In [22]:
model = keras.models.Sequential()
model.add(keras.layers.Conv1D(filters=256, kernel_size=5, padding='same', activation='relu', input_shape=(df_np_data.shape[1], 1)))
model.add(keras.layers.GlobalMaxPool1D())
model.add(keras.layers.Dense(units=df_np_data.shape[1], activation='linear'))
model.compile(optimizer='adam', loss='mean_squared_error', metrics=['mean_absolute_error'])
model.fit(x=Xin, y=Yin,batch_size=8, epochs=100,verbose=2 ,validation_split=0.1)
Train on 31 samples, validate on 4 samples
Epoch 1/100
 - 0s - loss: 0.0288 - mean_absolute_error: 0.1101 - val_loss: 0.4725 - val_mean_absolute_error: 0.6402
Epoch 2/100
 - 0s - loss: 0.0241 - mean_absolute_error: 0.0943 - val_loss: 0.4025 - val_mean_absolute_error: 0.5828
Epoch 3/100
 - 0s - loss: 0.0188 - mean_absolute_error: 0.0784 - val_loss: 0.3322 - val_mean_absolute_error: 0.5192
Epoch 4/100
 - 0s - loss: 0.0141 - mean_absolute_error: 0.0637 - val_loss: 0.2646 - val_mean_absolute_error: 0.4513
Epoch 5/100
 - 0s - loss: 0.0105 - mean_absolute_error: 0.0534 - val_loss: 0.2095 - val_mean_absolute_error: 0.3901
Epoch 6/100
 - 0s - loss: 0.0091 - mean_absolute_error: 0.0528 - val_loss: 0.1689 - val_mean_absolute_error: 0.3424
Epoch 7/100
 - 0s - loss: 0.0086 - mean_absolute_error: 0.0568 - val_loss: 0.1468 - val_mean_absolute_error: 0.3156
Epoch 8/100
 - 0s - loss: 0.0085 - mean_absolute_error: 0.0597 - val_loss: 0.1349 - val_mean_absolute_error: 0.3016
Epoch 9/100
 - 0s - loss: 0.0078 - mean_absolute_error: 0.0586 - val_loss: 0.1340 - val_mean_absolute_error: 0.3007
Epoch 10/100
 - 0s - loss: 0.0071 - mean_absolute_error: 0.0553 - val_loss: 0.1337 - val_mean_absolute_error: 0.3006
Epoch 11/100
 - 0s - loss: 0.0064 - mean_absolute_error: 0.0512 - val_loss: 0.1375 - val_mean_absolute_error: 0.3050
Epoch 12/100
 - 0s - loss: 0.0059 - mean_absolute_error: 0.0480 - val_loss: 0.1366 - val_mean_absolute_error: 0.3041
Epoch 13/100
 - 0s - loss: 0.0054 - mean_absolute_error: 0.0462 - val_loss: 0.1255 - val_mean_absolute_error: 0.2912
Epoch 14/100
 - 0s - loss: 0.0050 - mean_absolute_error: 0.0453 - val_loss: 0.1118 - val_mean_absolute_error: 0.2752
Epoch 15/100
 - 0s - loss: 0.0044 - mean_absolute_error: 0.0443 - val_loss: 0.1034 - val_mean_absolute_error: 0.2651
Epoch 16/100
 - 0s - loss: 0.0043 - mean_absolute_error: 0.0442 - val_loss: 0.0950 - val_mean_absolute_error: 0.2544
Epoch 17/100
 - 0s - loss: 0.0038 - mean_absolute_error: 0.0434 - val_loss: 0.0923 - val_mean_absolute_error: 0.2509
Epoch 18/100
 - 0s - loss: 0.0035 - mean_absolute_error: 0.0427 - val_loss: 0.0915 - val_mean_absolute_error: 0.2494
Epoch 19/100
 - 0s - loss: 0.0032 - mean_absolute_error: 0.0416 - val_loss: 0.0923 - val_mean_absolute_error: 0.2501
Epoch 20/100
 - 0s - loss: 0.0029 - mean_absolute_error: 0.0403 - val_loss: 0.0930 - val_mean_absolute_error: 0.2506
Epoch 21/100
 - 0s - loss: 0.0028 - mean_absolute_error: 0.0400 - val_loss: 0.0940 - val_mean_absolute_error: 0.2515
Epoch 22/100
 - 0s - loss: 0.0026 - mean_absolute_error: 0.0391 - val_loss: 0.0946 - val_mean_absolute_error: 0.2522
Epoch 23/100
 - 0s - loss: 0.0025 - mean_absolute_error: 0.0385 - val_loss: 0.0957 - val_mean_absolute_error: 0.2536
Epoch 24/100
 - 0s - loss: 0.0024 - mean_absolute_error: 0.0381 - val_loss: 0.0979 - val_mean_absolute_error: 0.2561
Epoch 25/100
 - 0s - loss: 0.0023 - mean_absolute_error: 0.0375 - val_loss: 0.1007 - val_mean_absolute_error: 0.2592
Epoch 26/100
 - 0s - loss: 0.0022 - mean_absolute_error: 0.0369 - val_loss: 0.1029 - val_mean_absolute_error: 0.2618
Epoch 27/100
 - 0s - loss: 0.0022 - mean_absolute_error: 0.0365 - val_loss: 0.1052 - val_mean_absolute_error: 0.2645
Epoch 28/100
 - 0s - loss: 0.0021 - mean_absolute_error: 0.0361 - val_loss: 0.1063 - val_mean_absolute_error: 0.2657
Epoch 29/100
 - 0s - loss: 0.0020 - mean_absolute_error: 0.0356 - val_loss: 0.1072 - val_mean_absolute_error: 0.2669
Epoch 30/100
 - 0s - loss: 0.0020 - mean_absolute_error: 0.0352 - val_loss: 0.1087 - val_mean_absolute_error: 0.2687
Epoch 31/100
 - 0s - loss: 0.0020 - mean_absolute_error: 0.0348 - val_loss: 0.1102 - val_mean_absolute_error: 0.2706
Epoch 32/100
 - 0s - loss: 0.0020 - mean_absolute_error: 0.0344 - val_loss: 0.1117 - val_mean_absolute_error: 0.2724
Epoch 33/100
 - 0s - loss: 0.0019 - mean_absolute_error: 0.0341 - val_loss: 0.1127 - val_mean_absolute_error: 0.2737
Epoch 34/100
 - 0s - loss: 0.0019 - mean_absolute_error: 0.0341 - val_loss: 0.1132 - val_mean_absolute_error: 0.2743
Epoch 35/100
 - 0s - loss: 0.0019 - mean_absolute_error: 0.0338 - val_loss: 0.1139 - val_mean_absolute_error: 0.2753
Epoch 36/100
 - 0s - loss: 0.0019 - mean_absolute_error: 0.0335 - val_loss: 0.1144 - val_mean_absolute_error: 0.2761
Epoch 37/100
 - 0s - loss: 0.0018 - mean_absolute_error: 0.0332 - val_loss: 0.1146 - val_mean_absolute_error: 0.2763
Epoch 38/100
 - 0s - loss: 0.0018 - mean_absolute_error: 0.0331 - val_loss: 0.1147 - val_mean_absolute_error: 0.2764
Epoch 39/100
 - 0s - loss: 0.0018 - mean_absolute_error: 0.0330 - val_loss: 0.1147 - val_mean_absolute_error: 0.2765
Epoch 40/100
 - 0s - loss: 0.0018 - mean_absolute_error: 0.0328 - val_loss: 0.1149 - val_mean_absolute_error: 0.2768
Epoch 41/100
 - 0s - loss: 0.0018 - mean_absolute_error: 0.0326 - val_loss: 0.1153 - val_mean_absolute_error: 0.2773
Epoch 42/100
 - 0s - loss: 0.0018 - mean_absolute_error: 0.0323 - val_loss: 0.1158 - val_mean_absolute_error: 0.2781
Epoch 43/100
 - 0s - loss: 0.0018 - mean_absolute_error: 0.0322 - val_loss: 0.1158 - val_mean_absolute_error: 0.2781
Epoch 44/100
 - 0s - loss: 0.0017 - mean_absolute_error: 0.0322 - val_loss: 0.1158 - val_mean_absolute_error: 0.2780
Epoch 45/100
 - 0s - loss: 0.0017 - mean_absolute_error: 0.0320 - val_loss: 0.1157 - val_mean_absolute_error: 0.2781
Epoch 46/100
 - 0s - loss: 0.0017 - mean_absolute_error: 0.0319 - val_loss: 0.1158 - val_mean_absolute_error: 0.2782
Epoch 47/100
 - 0s - loss: 0.0017 - mean_absolute_error: 0.0316 - val_loss: 0.1160 - val_mean_absolute_error: 0.2787
Epoch 48/100
 - 0s - loss: 0.0017 - mean_absolute_error: 0.0314 - val_loss: 0.1161 - val_mean_absolute_error: 0.2788
Epoch 49/100
 - 0s - loss: 0.0017 - mean_absolute_error: 0.0313 - val_loss: 0.1161 - val_mean_absolute_error: 0.2789
Epoch 50/100
 - 0s - loss: 0.0017 - mean_absolute_error: 0.0311 - val_loss: 0.1161 - val_mean_absolute_error: 0.2788
Epoch 51/100
 - 0s - loss: 0.0017 - mean_absolute_error: 0.0310 - val_loss: 0.1160 - val_mean_absolute_error: 0.2787
Epoch 52/100
 - 0s - loss: 0.0016 - mean_absolute_error: 0.0309 - val_loss: 0.1160 - val_mean_absolute_error: 0.2787
Epoch 53/100
 - 0s - loss: 0.0016 - mean_absolute_error: 0.0308 - val_loss: 0.1159 - val_mean_absolute_error: 0.2786
Epoch 54/100
 - 0s - loss: 0.0016 - mean_absolute_error: 0.0306 - val_loss: 0.1159 - val_mean_absolute_error: 0.2787
Epoch 55/100
 - 0s - loss: 0.0016 - mean_absolute_error: 0.0304 - val_loss: 0.1160 - val_mean_absolute_error: 0.2788
Epoch 56/100
 - 0s - loss: 0.0016 - mean_absolute_error: 0.0303 - val_loss: 0.1159 - val_mean_absolute_error: 0.2787
Epoch 57/100
 - 0s - loss: 0.0016 - mean_absolute_error: 0.0302 - val_loss: 0.1160 - val_mean_absolute_error: 0.2790
Epoch 58/100
 - 0s - loss: 0.0016 - mean_absolute_error: 0.0301 - val_loss: 0.1162 - val_mean_absolute_error: 0.2793
Epoch 59/100
 - 0s - loss: 0.0016 - mean_absolute_error: 0.0299 - val_loss: 0.1161 - val_mean_absolute_error: 0.2791
Epoch 60/100
 - 0s - loss: 0.0015 - mean_absolute_error: 0.0297 - val_loss: 0.1162 - val_mean_absolute_error: 0.2792
Epoch 61/100
 - 0s - loss: 0.0015 - mean_absolute_error: 0.0295 - val_loss: 0.1167 - val_mean_absolute_error: 0.2797
Epoch 62/100
 - 0s - loss: 0.0015 - mean_absolute_error: 0.0294 - val_loss: 0.1168 - val_mean_absolute_error: 0.2799
Epoch 63/100
 - 0s - loss: 0.0015 - mean_absolute_error: 0.0293 - val_loss: 0.1165 - val_mean_absolute_error: 0.2796
Epoch 64/100
 - 0s - loss: 0.0015 - mean_absolute_error: 0.0292 - val_loss: 0.1163 - val_mean_absolute_error: 0.2794
Epoch 65/100
 - 0s - loss: 0.0015 - mean_absolute_error: 0.0290 - val_loss: 0.1164 - val_mean_absolute_error: 0.2796
Epoch 66/100
 - 0s - loss: 0.0015 - mean_absolute_error: 0.0288 - val_loss: 0.1167 - val_mean_absolute_error: 0.2799
Epoch 67/100
 - 0s - loss: 0.0015 - mean_absolute_error: 0.0286 - val_loss: 0.1165 - val_mean_absolute_error: 0.2797
Epoch 68/100
 - 0s - loss: 0.0015 - mean_absolute_error: 0.0286 - val_loss: 0.1167 - val_mean_absolute_error: 0.2798
Epoch 69/100
 - 0s - loss: 0.0014 - mean_absolute_error: 0.0286 - val_loss: 0.1168 - val_mean_absolute_error: 0.2799
Epoch 70/100
 - 0s - loss: 0.0014 - mean_absolute_error: 0.0283 - val_loss: 0.1171 - val_mean_absolute_error: 0.2804
Epoch 71/100
 - 0s - loss: 0.0014 - mean_absolute_error: 0.0281 - val_loss: 0.1172 - val_mean_absolute_error: 0.2805
Epoch 72/100
 - 0s - loss: 0.0014 - mean_absolute_error: 0.0279 - val_loss: 0.1174 - val_mean_absolute_error: 0.2807
Epoch 73/100
 - 0s - loss: 0.0014 - mean_absolute_error: 0.0279 - val_loss: 0.1172 - val_mean_absolute_error: 0.2805
Epoch 74/100
 - 0s - loss: 0.0014 - mean_absolute_error: 0.0278 - val_loss: 0.1173 - val_mean_absolute_error: 0.2808
Epoch 75/100
 - 0s - loss: 0.0014 - mean_absolute_error: 0.0276 - val_loss: 0.1176 - val_mean_absolute_error: 0.2811
Epoch 76/100
 - 0s - loss: 0.0014 - mean_absolute_error: 0.0274 - val_loss: 0.1176 - val_mean_absolute_error: 0.2811
Epoch 77/100
 - 0s - loss: 0.0014 - mean_absolute_error: 0.0274 - val_loss: 0.1177 - val_mean_absolute_error: 0.2811
Epoch 78/100
 - 0s - loss: 0.0014 - mean_absolute_error: 0.0274 - val_loss: 0.1180 - val_mean_absolute_error: 0.2814
Epoch 79/100
 - 0s - loss: 0.0013 - mean_absolute_error: 0.0271 - val_loss: 0.1179 - val_mean_absolute_error: 0.2815
Epoch 80/100
 - 0s - loss: 0.0013 - mean_absolute_error: 0.0270 - val_loss: 0.1180 - val_mean_absolute_error: 0.2815
Epoch 81/100
 - 0s - loss: 0.0013 - mean_absolute_error: 0.0269 - val_loss: 0.1183 - val_mean_absolute_error: 0.2820
Epoch 82/100
 - 0s - loss: 0.0013 - mean_absolute_error: 0.0267 - val_loss: 0.1184 - val_mean_absolute_error: 0.2822
Epoch 83/100
 - 0s - loss: 0.0013 - mean_absolute_error: 0.0266 - val_loss: 0.1184 - val_mean_absolute_error: 0.2822
Epoch 84/100
 - 0s - loss: 0.0013 - mean_absolute_error: 0.0265 - val_loss: 0.1187 - val_mean_absolute_error: 0.2825
Epoch 85/100
 - 0s - loss: 0.0013 - mean_absolute_error: 0.0266 - val_loss: 0.1185 - val_mean_absolute_error: 0.2822
Epoch 86/100
 - 0s - loss: 0.0013 - mean_absolute_error: 0.0265 - val_loss: 0.1188 - val_mean_absolute_error: 0.2825
Epoch 87/100
 - 0s - loss: 0.0013 - mean_absolute_error: 0.0262 - val_loss: 0.1187 - val_mean_absolute_error: 0.2825
Epoch 88/100
 - 0s - loss: 0.0013 - mean_absolute_error: 0.0260 - val_loss: 0.1193 - val_mean_absolute_error: 0.2832
Epoch 89/100
 - 0s - loss: 0.0013 - mean_absolute_error: 0.0260 - val_loss: 0.1197 - val_mean_absolute_error: 0.2836
Epoch 90/100
 - 0s - loss: 0.0013 - mean_absolute_error: 0.0259 - val_loss: 0.1196 - val_mean_absolute_error: 0.2836
Epoch 91/100
 - 0s - loss: 0.0012 - mean_absolute_error: 0.0258 - val_loss: 0.1196 - val_mean_absolute_error: 0.2836
Epoch 92/100
 - 0s - loss: 0.0012 - mean_absolute_error: 0.0256 - val_loss: 0.1196 - val_mean_absolute_error: 0.2836
Epoch 93/100
 - 0s - loss: 0.0012 - mean_absolute_error: 0.0256 - val_loss: 0.1195 - val_mean_absolute_error: 0.2833
Epoch 94/100
 - 0s - loss: 0.0012 - mean_absolute_error: 0.0256 - val_loss: 0.1197 - val_mean_absolute_error: 0.2837
Epoch 95/100
 - 0s - loss: 0.0012 - mean_absolute_error: 0.0254 - val_loss: 0.1202 - val_mean_absolute_error: 0.2844
Epoch 96/100
 - 0s - loss: 0.0012 - mean_absolute_error: 0.0253 - val_loss: 0.1202 - val_mean_absolute_error: 0.2844
Epoch 97/100
 - 0s - loss: 0.0012 - mean_absolute_error: 0.0252 - val_loss: 0.1203 - val_mean_absolute_error: 0.2845
Epoch 98/100
 - 0s - loss: 0.0012 - mean_absolute_error: 0.0252 - val_loss: 0.1204 - val_mean_absolute_error: 0.2845
Epoch 99/100
 - 0s - loss: 0.0012 - mean_absolute_error: 0.0251 - val_loss: 0.1204 - val_mean_absolute_error: 0.2845
Epoch 100/100
 - 0s - loss: 0.0012 - mean_absolute_error: 0.0250 - val_loss: 0.1203 - val_mean_absolute_error: 0.2844
Out[22]:
<tensorflow.python.keras.callbacks.History at 0x189b799c208>

One we sucessfully train a validated NN with good performance, we can deploy the model into google cloud in order to be able to request predictions from any computer. In order to acoplish this we must follow the steps:

  1. Google cloud only accepts models created in SckitLear, XGboost and TensorFlow. Given that our neural network was created using the library Keras with a TensorFlow backend our first steps must be convert the Keras model into a TensorFlow SavedModel
In [23]:
model_input_name = model.input_names[0]
estimator_model = keras.estimator.model_to_estimator(keras_model=model, model_dir="./estimator_model/")
INFO:tensorflow:Using default config.
INFO:tensorflow:Using default config.
INFO:tensorflow:Using the Keras model provided.
INFO:tensorflow:Using the Keras model provided.
INFO:tensorflow:Using config: {'_model_dir': './estimator_model/', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': allow_soft_placement: true
graph_options {
  rewrite_options {
    meta_optimizer_iterations: ONE
  }
}
, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x00000189B41DB588>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}
INFO:tensorflow:Using config: {'_model_dir': './estimator_model/', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': allow_soft_placement: true
graph_options {
  rewrite_options {
    meta_optimizer_iterations: ONE
  }
}
, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x00000189B41DB588>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}

This code will create a folder named Keras inside the apointed directory. We need to take the files out of that folder and into the root apointed directory.

  1. Now we need to create a function that tells the model the kind of inputs it recieves. In this case it needs to be a time series of 210 records, because that was the input for the previously trained NN
In [24]:
def serving_input_receiver_fn():
    Series = {model_input_name:tf.placeholder(dtype=tf.float32,name='Time_Series',shape=[None,210,1]),}
    return tf.estimator.export.ServingInputReceiver(Series, Series)
  1. Now we export the model as a .pb (TensorFlow Extension)
In [25]:
export_path = estimator_model.export_savedmodel('./', serving_input_receiver_fn=serving_input_receiver_fn)
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Signatures INCLUDED in export for Classify: None
INFO:tensorflow:Signatures INCLUDED in export for Classify: None
INFO:tensorflow:Signatures INCLUDED in export for Regress: None
INFO:tensorflow:Signatures INCLUDED in export for Regress: None
INFO:tensorflow:Signatures INCLUDED in export for Predict: ['serving_default']
INFO:tensorflow:Signatures INCLUDED in export for Predict: ['serving_default']
INFO:tensorflow:Signatures INCLUDED in export for Train: None
INFO:tensorflow:Signatures INCLUDED in export for Train: None
INFO:tensorflow:Signatures INCLUDED in export for Eval: None
INFO:tensorflow:Signatures INCLUDED in export for Eval: None
INFO:tensorflow:Restoring parameters from ./estimator_model/keras_model.ckpt
INFO:tensorflow:Restoring parameters from ./estimator_model/keras_model.ckpt
WARNING:tensorflow:From C:\Users\DELL\AppData\Local\Continuum\anaconda3\lib\site-packages\tensorflow\python\estimator\estimator.py:1044: calling SavedModelBuilder.add_meta_graph_and_variables (from tensorflow.python.saved_model.builder_impl) with legacy_init_op is deprecated and will be removed in a future version.
Instructions for updating:
Pass your op to the equivalent parameter main_op instead.
WARNING:tensorflow:From C:\Users\DELL\AppData\Local\Continuum\anaconda3\lib\site-packages\tensorflow\python\estimator\estimator.py:1044: calling SavedModelBuilder.add_meta_graph_and_variables (from tensorflow.python.saved_model.builder_impl) with legacy_init_op is deprecated and will be removed in a future version.
Instructions for updating:
Pass your op to the equivalent parameter main_op instead.
INFO:tensorflow:Assets added to graph.
INFO:tensorflow:Assets added to graph.
INFO:tensorflow:No assets to write.
INFO:tensorflow:No assets to write.
INFO:tensorflow:SavedModel written to: .\temp-b'1547486173'\saved_model.pb
INFO:tensorflow:SavedModel written to: .\temp-b'1547486173'\saved_model.pb

This will create a folder with: A file called SavedModel.pb, that contains the model, and a folder caller variables, that contains the weights of the trained network

In [26]:
i = Image(filename='C:/Users/DELL/Pictures/Folder.PNG')
i
Out[26]:
  1. The files created in step 3 need to be uploaded to a bucket in google cloud storage
In [27]:
i = Image(filename='C:/Users/DELL/Pictures/Bucket.PNG')
i
Out[27]:
  1. We now go to de machine learning engine and create a model and a version
In [28]:
i = Image(filename='C:/Users/DELL/Pictures/Model.PNG')
i
Out[28]:
In [29]:
i = Image(filename='C:/Users/DELL/Pictures/Version.PNG')
i
Out[29]:
In [30]:
i = Image(filename='C:/Users/DELL/Pictures/VersionC.PNG')
i
Out[30]:
  1. Now that we have created our model in GCP we can test the predictions. For this, first we need to create a JSON file containing the time series. As an exaple we are gonna take the 24 th series.
In [31]:
Dat=df_np_data[24:25]
Dat=Dat.T
Dat=Dat.tolist()
Predictions={model_input_name:Dat}
json_filename = 'Serie24_Def_1.json'
with open(json_filename, 'w') as outfile:
    json.dump(Predictions, outfile)
  1. Finally we get to Google Cloud Shell and excecute the following comands

We define the model name, the version and the input file (This file needs to be uploades to shell first)

MODEL_NAME=AnomalyDetection

INPUT_DATA_FILE="Serie24_Def_1.json"

VERSION_NAME=RealAD

In [32]:
i = Image(filename='C:/Users/DELL/Pictures/Shell.PNG')
i
Out[32]:

With the parameters defined we request some predictions with the code

In [34]:
i = Image(filename='C:/Users/DELL/Pictures/Pred.PNG')
i
Out[34]:

After runnig the code the requested predictions appear on screen

In [35]:
i = Image(filename='C:/Users/DELL/Pictures/Predd.PNG')
i
Out[35]: